RLWM ACT-R log-likelihood model fitting

Abstract

Abstract goes here

Model fit analysis

2 x 2 setzise by iteration(learn vs test) ANOVA table for behavioral data
term	df	sumsq	meansq	statistic	p.value
setSize	1	0.0003843	0.0003843	0.0292196	0.8642903
iteration	1	7.7448344	7.7448344	588.7892899	0.0000000
setSize:iteration	1	0.6490973	0.6490973	49.3466348	0.0000000
Residuals	1988	26.1498145	0.0131538	NA	NA

2 x 2 setzise by iteration(learn vs test) ANOVA table for model data
term	df	sumsq	meansq	statistic	p.value
setSize	1	0.1612980	0.1612980	15.49667	8.55e-05
iteration	1	9.5609337	9.5609337	918.56458	0.00e+00
setSize:iteration	1	0.7419769	0.7419769	71.28527	0.00e+00
Residuals	1988	20.6922155	0.0104086	NA	NA

RMSE vs Log-Likelihood: which method resulted in better a fit?

The log-likelihood procedure resulted in more parameter-sets fitting participants over all:

log-likelihood: 50 unique parameter-sets (biased:36, LTM:20, meta_RL:2, RL:0)

RMSE: 37 unique parameter-sets (biased:11, LTM:15, meta_RL:9, RL:2)
According to the figure below it seems like the differences between the fits of the two models are very small with the RMSE method potentially being a better fit, especially for set size 6 data.

Correlation b/n behavioral and model data from the two procedures:

#> 
#>  Welch Two Sample t-test
#> 
#> data:  corr by method
#> t = 6.3954, df = 317.1, p-value = 5.735e-10
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  0.04212052 0.07955197
#> sample estimates:
#> mean in group rmse mean in group logL 
#>          0.9046639          0.8438277

Looking at the violin plots above, data generated by the RMSE method is more strongly correlated (mean r: 0.9046639) with the behavioral data than those from the Log-Likelihood data (mean r: 0.8438277 ).

RMSE analysis

t-test on mean squared errors
estimate	estimate1	estimate2	statistic	p.value	parameter	conf.low	conf.high	method	alternative
0.0374	0.128	0.0905	8.61	0	149	0.0288	0.046	Welch Two Sample t-test	two.sided

The log-likelihood method had lower errors (RMSE: 0.0904838) than the RMSE method (RMSE: 0.1278827) when comparing fits of the selected models to behavioral data by the two methods. This difference is significant as determined by a t-test.

BIC Analysis

Statistics on where the next best fit occurs for each participant by model
model	mean	median	sd	min	max
biased	43.2459	8	119.99204	2	657
LTM	13.5000	9	15.20561	2	48
metaRL	3.0000	3	0.00000	3	3

These figures (specifically BIC fig 2) show that the best fit model has strong evidence against the second best fit model.These fits are better compared to the RMSE method (see RMSE document for comparison).

BIC fig 1

BIC Fig 2

BIC fig 3

These subjects had second best fit models that came from a different model group. X1 to X3 are the BIC differences.
subjects	X1	X2	X3	model
6218	1.6800691	0.7662433	0.0131273	LTM
6226	20.7272476	0.1642479	1.5106469	LTM
6234	17.4765377	0.4880064	0.3893591	LTM
15005	26.7061902	6.8416407	8.8128153	biased
15006	14.9020846	17.2172782	3.6532177	biased
15007	26.5035268	0.2165939	5.6548364	biased
15012	22.6166963	3.6827696	6.3863653	LTM
15020	21.0249436	7.4761139	4.9440946	LTM
28307	4.3697057	2.1578692	3.2079443	biased
29245	0.0178342	7.1627899	3.3654804	biased

Additional metrics

Figure 11.

#> 
#>  Welch Two Sample t-test
#> 
#> data:  estimate by setSize
#> t = 10.149, df = 142.26, p-value < 2.2e-16
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#>  0.02799973 0.04154511
#> sample estimates:
#> mean in group s3 mean in group s6 
#>       0.11481641       0.08004399

Descriptive stats of model and behavioral learning rate
setSize	type	model	mean	se
s3	behav	biased	0.1163739	0.0021718
s3	behav	LTM	0.1130357	0.0036576
s3	behav	metaRL	0.0851190	0.0017857
s3	model	biased	0.1141015	0.0032627
s3	model	LTM	0.1131286	0.0012347
s3	model	metaRL	0.0973333	0.0065714
s6	behav	biased	0.0792870	0.0036193
s6	behav	LTM	0.0826984	0.0043388
s6	behav	metaRL	0.0765873	0.0115079
s6	model	biased	0.0849828	0.0022839
s6	model	LTM	0.1022738	0.0017649
s6	model	metaRL	0.0829286	0.0030714

Figure 12

K-W test one way rank-sum test: s6-s3 learning curve differences by model type for behavioral data
statistic	p.value	parameter	method
193.4056	0	2	Kruskal-Wallis rank sum test

K-W test one way rank-sum test: s6-s3 learning curve differences by model type for model data
statistic	p.value	parameter	method
391.6311	0	2	Kruskal-Wallis rank sum test

pairwise post-hoc tests for behav data
group1	group2	p.value
LTM	biased	0
metaRL	biased	0
metaRL	LTM	0

pairwise post-hoc tests for model data
group1	group2	p.value
LTM	biased	0
metaRL	biased	0
metaRL	LTM	0

Figure 13.

(2)set-size x (2)type(modelorBehav data) x 3(model) anova. RL excluded.
term	df	sumsq	meansq	statistic	p.value
setSize	1	5.5580884	5.5580884	438.4466841	0.0000000
type	1	0.1911125	0.1911125	15.0757993	0.0001050
model	2	1.0508771	0.5254386	41.4489263	0.0000000
setSize:type	1	0.0062083	0.0062083	0.4897418	0.4840835
setSize:model	2	0.9516216	0.4758108	37.5340689	0.0000000
type:model	2	0.0831864	0.0415932	3.2810574	0.0376904
setSize:type:model	2	0.0648987	0.0324494	2.5597497	0.0774516
Residuals	3972	50.3521358	0.0126768	NA	NA

Parameter analysis

mean and medians of parameter values
variable	mean	median
alpha	0.1500000	0.1500000
egs	0.3000000	0.3000000
bll	0.5000000	0.5000000
imag	0.3000000	0.3000000
ans	0.3000000	0.3000000
strtg	0.4402444	0.3604394

Figure 15